monitor-freshness: alert when a monitor stops completing, not just when it fails - #276

Step 1 of #258, diagnosed in this comment.

The problem

rekor-monitor.yml last completed on 2026-08-21 and was cancelled on every run for the thirteen days after. Nothing said so.

The reusable workflow's notification job fires on failure. A cancelled run is not a failed one, so the if-failed job skipped every time and the Actions tab stayed green while the Sigstore identity monitoring — the control that workflow's own header credits with catching the Shai-Hulud worm — was doing nothing at all.

The generalisable lesson is that the status of the last run cannot tell you whether a monitor is working. A monitor can be cancelled, skipped, disabled, or have its schedule quietly dropped by GitHub, and in every one of those cases the newest row still looks recent. The one signal that survives all of them is the age of the last success, which is what this checks.

What this adds

• scripts/check-monitor-freshness.mjs — reads the newest successful run of each declared monitor and files one issue per stuck one. Dependency-free on purpose: the watcher must not be takeable down by the thing it would then fail to report. --dry-run prints the table and writes nothing.
• .github/workflows/monitor-freshness.yml — every 6 hours, matched to the tightest declared threshold so an alert is never more than one period behind the condition. contents: read by default, issues: write only on the job, both actions SHA-pinned.
• scripts/check-monitor-freshness.test.mjs — 21 tests, wired into npm run check.

Design notes worth review

An API failure is reported as unchecked, never as healthy. This has a regression test, because the first draft got it wrong: it swallowed a 401 and printed All monitors have completed within their thresholds, exit 0. That reproduced, inside the watcher, the exact bug the watcher exists to catch — absence of a signal read as the presence of health. A 404 is treated as a genuinely absent workflow and skipped; everything else counts as unchecked and turns the run red.

Thresholds are multiples of each monitor's own period, not round numbers. rekor-monitor is hourly and gets 6h, so five dropped or overrunning slots pass quietly; link-check is weekly and gets 10 days. GitHub drops scheduled runs under load, so a threshold at 1× the period would be noise, and noise is how a real alert gets ignored.

One issue per stuck monitor, then silence. A watcher that opens a fresh issue every six hours is its own kind of silence. The title is stable per monitor and an existing open issue suppresses re-filing.

The clock and fetch are injected, so every decision path is exercised offline against a stubbed API. A monitor's own tests must not be hostage to the service it monitors — the same reasoning link-check.yml's header already records for keeping external probes out of the build.

Not part of npm run check. That suite is hermetic and this needs the network. Its tests are in check; the check itself is a schedule, like the monitors it watches.

Verification

$ node --test scripts/legibility/coldread.test.mjs scripts/check-monitor-freshness.test.mjs
# tests 51
# pass 51
# fail 0

check-node-uniqueness and check-license both still pass. I could not exercise the live API path from my environment — no usable token — which is part of why the decision path is covered by a stubbed suite rather than a smoke run.

Out of scope, still open on #258

Deliberately not attempted here, because both need a maintainer's judgement about the security posture:

• Re-seeding the checkpoint. The gap was already ~201.6M log entries on 08-29 and grows hourly. It needs one long dispatch with nothing competing for the slot.
• The cadence. Today's run stopped at 59m57s, about 20 seconds before the next 41 * * * * tick. That points at concurrency cancellation rather than a timeout, since GitHub's default job timeout is 360 minutes, but I could not confirm it from the logs alone. If it is concurrency, the scan simply takes longer than the interval between scans and the interval has to move.

Note that this PR does not make rekor-monitor work. It makes the next thirteen-day outage visible on day one. Given that, Claim-issue: rather than Closes: — #258 stays open.

Also worth considering, though not done here: this is generic enough to belong in bounded-systems/.github as a reusable that every repo calls, the shape claim-sweep and pr-claim already use. Kept to one repo until it has earned that.

Claim-issue: #258

🤖 Generated with Claude Code

https://claude.ai/code/session_01FQNTis6LuJHd5KDXTvYG74

Generated by Claude Code